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(54) Packet switched communication system and traffic shaping process 



(57) A packet switched communication system and, 
more particularly, a traffic shaping process for causing 
time multiplexed packet flows at queuing points within 
such systems on system elements to conform to spec- 



ified traffic descriptors. The invention provides rate 
shaping in per-flow output queued routing mechanisms 
for available bit rate (ABR) service in networks having 
segmented ABR control loops (102). 
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Description 

The present invention relates to packet switched communication systems and, more particularly, to traffic shaping 
for causing the time multiplexed packet flows at queuing points within such systems or system elements to conform 
to specified traffic descriptors. 

A. Traffic Contracts/Definition* 

Most applications that are currently running on packet switched communication networks can function acceptably 
with whatever bandwidth they happen to obtain from the network because they have 'elastic- bandwidth requirements^ 
The service classes that support these applications is known as 'best efforts' service in the Internet community and 
as 'Available Bit Rate" (ABR) in the Broadband ISDN/ATM community. 

There is however a growing demand for network services that provide bounded jitter or, in other words, bounded 
packet delay varation (common* referred to as cell delay variation in an ATM context). For example, the type . of 
service is required for real time applications, such as circuit emulation and video. It is nol clear whether and how the 
internet community will respond to this demand, but the Broadband ISDN/ATM community has responded by introduc- 
ina the notion of a user-network negotiated traffic contract. 

As is known a user-netwoifc ATM contract is defined by a traffic descriptor which includes traffic parameters, 
tolerances and quality of service requirements. A conformance definitton is specified for each of the relevant traffic 
parameters Accordingly, ATM services may make use of these traffic parameters and their corresponding conformance 
specifications to support different combinations of Quality of Service (QoS) objectives and multiplexing schemes. 
Partially overlapping sets of ATM traffic classes have been defined by the Telecommunications Standardization Sector 
of the International Telecommuncations Union (ITU-T) and the ATM Forum. In some instances, traffic classes which 
have essentially identical attributes have been given different names by these two groups, so the foltowmg name 
translation table identifies the existing equivalent counterparts: 



ITU-T Traffic Class 


ATM Forum Traffic Class 


ABR 

Deterministic Bit Rate (DBR) 
Statistical Bit Rate (SBR) 
(No existing counterpart, but understudy) 
(No existing counterpart, but understudy) 


ABR 

Constant Bit Rate (CBR) 
Variable Bit Rate (VBR) 
Real time Variable Bit Rate (rt-VBR) 
Unspecified Bit Rate (UBR) 



An ATM serv.ce contract tor a virtual circuit (VC) connection or a virtual path (VP) connection rnay include multiple 
Daram^rerTdesc'bing the service rate of the connection. This includes the Peak Cell Rate (PCR). the Sustainable 
K^sS^nSii Burst Tolerance (IBT), and the Minimum Cel. Rate (MCR). Not all of these parameters 
are Sam for every connection or every service class, but when they are .np.ied or explcitly specjed elements o 
me se^ce contract they must be respected. VC connections are the primary focus of the following discussion, but « 
Z SSSStlvP connections^ also be so specified. The data transport unrt for an ATM 
Ts elerred to as a 'cell.' In thte disclosure, however, the term 'packet' sometime • used to '^^^anspon 
un t because this more general terminology is consistent with some of the broader aspects of the innovations. ^ 

X Genenc Ceil Plte Algorithm (GCRA), which is specified in t^T R^M 
testina a oacket or cell flow for conformance with a traffic descriptor. To perform such testing, the GCRA requ res the 
Mission interval (i.e.. the reciprocal of a flow rate) and a tolerance. , In practice, this tolerance 
may depend on a variety of factors, including me cx^nert 

Si be see^, the GCRA can be employed as a Boolean function, where for a flow of foced «■ packe s o cells on 
a connects the GCRA (emission interval, to.erance) is false if me flow . «*m «> - P-* J£J r tm. h «tow 
is conforming to a m.nimum rate. For example, a source of cells conforms to a PCR if GCRA O^R. W » ™* 
Likewise a connectionorflowconformsto a nMCRifGCRA(l/MCR,x MCR )isfalse.Asw,llbeapprec,atedthe em.ss.on 

m Z%^^TXZ^o, a source which establishes a connection in the expectation that astatic 
amount of bandwidth will be Sinuously available to the connection throughout «s lifetime. Thus, the bandw.dth the 
network c £S» To a DBR connection is characterized by a PCR value. Further, the cel. or P^^.^w on s ch a 
^ct,o^omp,ies with me traffic contract ■ it conforms to GCRA (t/PCR. t^) On me ^er h^d an SBR traffic 
comract is suitable for an application which has known traffic characteristics mat allow for an informed selects of an 
SCRTndV B T " s «■ as a PCR and x rcR . An SBR or rt-SBR flow comp.ies with its traffic contract rf the flow not onry 
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conforms to GCRA (1/PCR, tpcg), but also to GCRA (1/SCR, t, bt ). 

As previously indicated, an ABR traffic contract is appropriate for applications that can tolerate the dynamic vari- 
ations in the information transfer rate that result from the use of unreserved bandwidth. A PCR and an MCR are specified 
by the source establishing such a connection, and these parameters may be subject to negotiation with the network. 
5 Thus, the bandwidth that is available on an ABR connection is the sum of the MCR (which can be 0) and a variable 
cell rate that results form a sharing of unreserved bandwidth among ABR connections via a defined allocation policy 
(i.e., the bandwidth a source receives above its specified MCR depends not only on the negotiated PCR, but also on 
network policy). Feedback from the network enables the source application to dynamically adjust the rate it feeds cells 
or packets into an ABR connection. An ABR flow always complies with its traffic contract if it conforms to GCRA (1/MCR, 
10 t mcr). and is always non-compliant if it does not conform to GCRA (1/PCR, tp^). Conformance in the region between 
MCR and PCR is dependent on the ABR feedback and is thus dynamically determined. 

A UBR traffic contract is similar to the ABR contract, except that the UBR contract does not accommodate the 
specification of an MCR and has no dynamic conformance definition. Therefore, a UBR flow complies with its traffic 
contract if it conforms to GCRA (1/PCR, Tpc R ). 

is 

B. Traffic Shaping 



ITU-T Recommendation 1 .371 addresses the possibility of reshaping traffic at a network element for the purpose 
of bringing the traffic into conformance with a traffic descriptor in the following terms: 
20 "Traffic shaping is a mechanism that alters the traffic characteristics of a stream of cells on a VCC or a VPC to 

achieve a desired modification of those traffic characteristics, in order to achieve better network efficiency whilst meeting 
the QoS objectives or to ensure conformance at a subsequent interface. Traffic shaping must maintain cell sequence 
integrity on an ATM connection. Shaping modifies traffic characteristics of a cell flow with the consequence of increasing 
the mean cell transfer delay. 

2S Examples of traffic shaping are peak cell rate reduction, burst length limiting, reduction of CDV by suitably spacing 

ceils in time and queue service schemes. 

It is a network operator's choice to determine whether and where traffic shaping is performed. As an example, a 
network operator may choose to perform traffic shaping in conjunction with suitable UPC/NPC functions 
It is an operator's option to perform traffic shaping on separate or aggregate cell flows. 
30 As a consequence, any ATM connection may be subject to traffic shaping. 

The options available to the network operator/service provider are the following : 



a. No shaping 

3S - Dimension the network in order to accommodate any flow of conforming cells at the network ingress whilst 

ensuring conformance at the network egress without any shaping function. 



b. Shaping 

*o - Dimension and operate the network so that any flow of conforming cells at the ingress is conveyed by the 

network or network segment whilst meeting QoS objectives and apply output shaping the traffic iri order to 
meet conformance tests at the egress. 

Shape the traffic at the ingress of the network or network segment and allocate resources according to the 
traffic characteristics achieved by shaping, whilst meeting QoS objectives and subsequent conformance tests 
45 at the network or network segment egress. 



Traffic shaping may also be used within the customer equipment or at the source in order to ensure that the cells 
generated by the source or at the UNI are conforming to the negotiated traffic contract relevant to the ATC that is used 
(see Section 5.5).* ITU-T Recommendation 1.371, Section 6.2.5. 

so 

C. Scheduling for Real Time and Non- Real Time Connection a/Ex fating Toola and Techniques 

As is known, if bandwidth is not divided fairly' between applications employing "best efforts" Internet service or 
ABR ATM service a variety of undesirable phenomena may occur. See Lefelhocz, Lyles, Shenker and Zhang, "Con- 
ss gestion Control for Best-Effort Service: Why we need a new paradigm,' IEEE Network . January/February 1996, for 
further details on mechanisms for best effort/ABR traffic. 

Most ATM switches currently are implemented with FIFO queuing. FIFO queuing exhibits pathological behaviors 
when used for ABR traffic (see "On Traffic Phase Effects in Packet-Switched Gateways" , Sally Floyd and van Jacobson, 
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i nternetworking Res**** and Experience, Vol. 3, pp. 115-156 (1992). and "Observations on the Dynamics of a Con- 

9? bSm September 3-6. 199V Zureh. Switzerland, pp. 133-148.). FIFO also is unable to protect correct* 
b havinTuse^ agaWst misbehaving users (I does not provide solatbn). As a result of these defines non-FIFO 

ysis and Simulation of a Fair Queuing Algorithm.- Proceedings of ACM Siqcornm, pages 1-12 September 1 »9j»d 
A K Parekh "A Generalized Processor Sharing Approach to Flow Control in Integrated Serv.ce Ne^orks. Ph D Thesis, 
Department o« Electrical Eng^ering and Computer Science. MIT, 1992.) or approxirmUons to * " 
round-robin (Ellen L Hahne, 'Round-robin Scheduling for Max-Min Fairness ti Data Networks, IEEE Journal on Se- 
..vtpriAr^s in Communications. Vol. 9. pp. 1024-1039, Sept. 1991.) are often suggested. 

Service classes which have inelastic bandwidth requirements often require that data be transmitted through the 
network with bounded jitter (i.e.. bounded cell or packet delay variation). As shown by the above referenced Parekh 
S wlhted fair queuing can be used to provide bounded jitter for real tin* streams. Moreover, Parekh's results 
S^rS^S^oyi Simon S. Lam and Hank* M. V.n, "Determining End-tc-End Delay Bounds .n ^teroge- 

SSSSa SgB^. ° umam - nh - «* 18 - 22 ' 199 * •> be en extended to prove ' delay J 00 , s a f T T 

IS the cSS elated mech anisms of Virtual Clock (Lccia Zhang, "Virtual C.ocfc A New Traffic Contra LAIgor hm for 
U r S y.l!!j- ,c. p^^inn SQ fAC M Siacomm. pa g esl9-29,August19^. )a ndSe.1<lockedFarO ueu . 

fng (tj GoSni, "A SeH-Ctocked Fair Queuing Scheme for High Speed Applications." Proceedings of INFOCOM, 

20 PP ^futt'is^ that both elastic (Best efforVABR) and inelastic (or real-time) services can benefit from the use 
of fair queuing and related algorithms. 

1. Weighted Fair Queuing and Virtual Clock 

Fairqueuing and related algorithms (e.g.. frame-based fair queuing, deficrt ^^^^^^^ 
«i nackats or other data transport units (e.g.. an ATM cell is a packet for the purposes of this discussion). For ATM 
ZZZ£Z^J!5^* either the' VC. or the VPI, while in the internet protocol suite the ^tonis on 
ma ba^s or<IP add eTs protoil. port> triples (IPv4) or flow identifiers (IPv6). In both seff<locked weighted fair 
a S ZEZ packets are ordered (sorted) by timestamps (schemes such as round-robin provde approx- 
Sns TJrTm of packets by stamps). These timestamps represent the virtual finish^ time 
»a^rtin g Sn. tar the packet and are computed by taking a starting time value ™*^™f] 
by multiplying the length of the packet by a weight which represents the P art.cular packet sequences share of the 

bandwidth. 4 . 
3S More particularly, for virtual clock the virtual finishing time is computed as. 

VT(f . 0) = 0 

VT(1. = max{Arrival(f, VT(f, j)} + Length(f, j+iyRate<f) . ( 1 ) 

where: VT(f , j) is the virtual finishing time associated with packet j of flow (virtual circuit) f; 
Arrival^ j) is the arrival time of packet j of flowf ; and 
Length(f, j) is the length of packet j of flow f 

Self-clocked weighted fair queuing assigns virtual finishing times according to the formula: 

VT(f, 0) = 0 

50 vT(f , j+1 ) = max{SystemVirtualTime. VT(f , j)} + Length(f j+1 ) * weight(f ) < 2 ) 

where- SystemVrfrtuamme is the virtual time associated with me packet being se.ed (being output^ ^ 

expression is: 
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VT(f, j+1 ) = max{Arrival(f, j+1 ), VT(f , j)} + constant(f) (3) 
For self-clocked weighted fair queuing, on the other hand, the simplified expression is' 

5 

VT(f, j+1 ) = max{SystemVirtualTime, VT(f, j)} + constant (0 (4) 

In other words, an ATM queuing point which implements either virtual clock or self<locked weighted fair queuing 
10 performs the following steps: 

1 ) compute the maximum of (a) the current virtual time for the VC, and (b) either of i) the arrival time of the cell or 
ii) the system virtual time. 

2) add to the results of step 1 above a per-VC constant representing that VC's share of the bandwidth. 

'5 3) service cells (transmit them) in order of increasing values of the virtual time stamps assigned by steps 1 and 2. 

2. Priority 

Giving priority to one traffic class over another means that if the higher priority traffic class has cells ready for 
20 transmission, those cells are always transmitted in preference to the cells of the lower priority traffic class. 

Priority mechanisms can be either preemptive or non-preemptive. This terminology comes from the operating 
system literature. A non-preemptive priority mechanism assigns a priority to an object (a process in the operating 
system world, a VC in the ATM world) at a scheduling time, and the object then retains this priority until it is served. 
Preemptive priority mechanisms, on the other hand, can change the priority of objects while they are waiting to be 
2S served. For example, in a preemptive system one could say 'schedule this VC with priority 3 but if it is not served within 
200 microseconds then increase its priority up to 2." 

3. Work Conserving and Non-work Conserving Queuing 

30 Kleinrock. Queuing Systems. Vol. 2: Computer Applications. John Wilev & Sons. N Y M Y 1QQ6 n 113 uses the 

terminology 'work conserving' to denote any queuing system in which work is neither created nor destroyed. In keeping 
with this terminology a switch which, when given queued cells, always transmits cells on the outgoing link is a 'work 
conserving switch'. Switches employing a pure FIFO, weighted fair queuing or virtual clock scheduling algorithm are 
all work conserving. In contrast, a non-work conserving switch may choose not to send cells, even when cells are 

3S queued for transmission. As will be seen, a method of doing this is to program the switch to wait until the current time 
is equal to or greater than the timestamp associated with a particular cell before transmitting that cell. 

Work conserving switches attempt to fully utilize the transmission link, but do not necessarily remove or prevent 
bursts. In contrast, non-work conserving switches can strategically delay cells so as to re-shape traffic to meet a more 
stringent conformance test (i.e., a GCRA with a smaller t). Additionally, a non-work conserving switch in which a given 

40 connection is only allocated a specified amount of buffering can perform a policing function (in ITU terms a UPC/NPC) 
by discarding or tagging cells which overflow the allotted buffer space. An example of a non-work conserving queuing 
system is the Stalled virtual Clock (Sugih Jamin, 'Stalled Virtual Clock" working note, Department of Computer Science, 
UCLA, March 21 , 1 994), which is an adaptation of Lixia Zhang's Virtual Clock algorithm where virtual time is not allowed 
to run faster (it stalls or goes non-work conserving) than real-time. Also see, work by Scott Shenker that is available 

45 by FTP at FTP.PARC.XEFOX.com. 

4. Calendar Queues 

A calendar queue is a time ordered list of actions, each of which is dequeued and executed when real-time is equal 
so to or greater than the time associated with the action. Calendar queues with bounded time intervals can be represented 
as a linear array which is known as a time-wheel' or time-line.' Time-wheels assign events to buckets relative to a 
pointer, where the bucket index is calculated using arithmetic modulo the wheel size. These data structures are well 
known in the literature as a queuing mechanism. In a time-wheel, absolute time is represented as an offset relative to 
the current time ("real time'), and each element in the array is a bucket which contains one or more actions (typically 
ss in linked-list form) which are to be executed at the time assigned to the bucket in which they reside. Any of the buckets 
of such a time-wheel can be empty, i.e., have no events associated with it. 

For every time-wheel, there are two times of interest: t ear1iMt and t^^ which correspond to the head and tail 
pointers for the active entries in the array; where is the time of the next entry (e.g. . packet or cell) to be serviced. 
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and t is the time associated with the latest (mostdistant in time) bucket containing a scheduled event. The difference 
and t tote . t stne time time-wheel, b, minus 1 Th.s can be ensured by 

Stte^JK " n^.o b and by then enLng that no offset (the packet length mujipjiec , * .JJjr 
ZTte or the weight in virtual clock or we.ghted fair queuing respectively) is greater than b-1 For an ATM link running 
rt (5 3 s^T 1 49 76 X - the SOnIt paytoad rate) there are approximately 353208 cells/sec on the link Ac- 
SSgvTe^bps (voce .euphony rates) flows (approximately 174 ce.ls/sec when AAL type 1 is used) are the .owes, 
snee7connectons that need to be supported, then the ratio of the highest supported rate to the lowest rate is 2029, 
which S oTL ratio ,s & maximum offset that will get added during the calculation o v«e^ 
Theref^TtiJe-whee. of length 2030 (2049 to allow for rounding up to a power of two) is sufficient to encode the 
wirt, <a i times associated with circuits ranging in rates from 64Kbps to full OC-3 link rate 

^^S^^SZna, can be decreased by permitting an array element to contain more than one «,me 
theibl/e^eLbed time-wheel is reduced to 256 elements from 2048. then each bucket wouK 
SeSJnTSSi "mapped No rt. Actions withr a single bucket that spans mu«iple offsets may be P^dart 
T^XZ^buc^xs^ will stay inorder. This reduces the amount of memory that needs to be allocated 
t ^'!SSSZ at the cost ot reducing the precise of me ordering of actons in the calendar queue. 

0. Traffic Shaping for Time Multiplexed Flow, on Multiple Output Channel. 

Preferably any traffic shaping that is needed to bring time muKiplexed packet or cell flows into conformance , with 

° Pti Howtver prk>r output queued ATM switches general* have employed FIFO (First In - First Out) out^-J 

wmmwmm 

these confltta cause ^ n *?™™^™™ ' to r traffic, such as DBR traffic, which general* 

^Th^e^ 



in which: 



45 in Fig. 1 ; 



so 



55 



Fi g Fi |isamorede,ai.edb.ockdia gra mof a representatr,e channel on the output or transm* side of the chip shown 
So*? schematically illustrates a stalled virtual clock calendar queue implementation o. the presenl l invenUon; 

fJ 7 SZS - ABR connection havfcg a ^ 

Fig. 8 schematically illustrates an ABR connection having a segmented control loop. 

A. A Representative Environment 
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physical layers that are connected to them, and to also exchange control cells with the control processor module 22. 
In keeping with standard practices, the communication channels are unidirectional, so a pair of channels are required 
for bi-directional communications. 

The switch 21 comprises a switching fabric 24. a fabric control module 25 and a reservation ring 26 for switching 

5 data and control cells from input queues to per-VC output queues. The cells in these queues are stored in the data 
path in data memory 27, and these input and output queues are managed by a queue control module 28. Typically, 
the data memory 27 is sized to hold up to roughly 1 2000 cells. Connection records for the data and control cell flows 
are stored in the control path in control memory 29, together with certain types of control cells which are intercepted 
by a rate based engine and traffic multiplexer 31 for routing to the control processor module 22. Suitably, the control 

io RAM 29 is capable of accommodating up to about 8200 connection records and 64k cell records. The interaction of 
the control processor module 22 with the switch 21 is beyond the scope of the present invention and, therefore, is not 
described herein. However, persons who are familiar with ATM switch design will understand that the control processor 
is primarily responsible for performing connection establishment and termination, as well as OAM (Operation and 
Maintenance) functions. 

*5 The data path of the switch 21 is synchronously clocked (by means not shown) at a predetermined rate of, say, 

40 MHz. However, in keeping with conventional synchronous pipeline design practices, the phase of this clock signal 
is delayed (by means also not shown) by differing amounts at different points along the data path to give the data 
adequate time to settle prior to being transferred from one pipelined stage to the next 

In accordance with standard practices, a source wishing to communicate with a destination initiates negotiations 

20 with the ATM network within which the switch 21 resides by sending a SETUP message to the network. This message 
identifies the destination and explicitly or impliedly specifies all of the relevant traffic parameters for the requested 
connection. If the network is prepared to commit to the traffic contract which is defined by these traffic parameters (or 
a modified version of the parameters that the source is willing to accept), the network routes the SETUP message to 
the destination. Then, if the destination is ready to receive message traffic from the source in accordance with the 

25 terms of the traffic contract, the destination returns a CONNECT message to the source. This CONNECT message 
confirms that a connection has been established on a specified virtual circuit (VC) within a specified virtual path (VP) 
for a cell flew that conforms to the traffic contract. See ITU-T Recommendation 0.2391 and ATM Forum UNI 4.0 Spec- 
ification. "Permanent" virtual connections can be established by provisioning, without invoking these signaling proto- 
cols. 

30 Data cells begin to flow after a connection is established. As shown in Fig. 2. the form of the cells change as they 

pass through the switch 21 because of the operations, the switch performs. Cells may be replicated within the switch 
21 for multicasting, but the following discussion will be limited to unicast operations to avoid unnecessary complexity. 

As indicated in Fig. 2 at 41 , each inbound cell that the switch 21 receives has a header containing a VP index and 
a VC index. These indices combine to define a unique address for one hop of the connection. A connection may be 

35 composed of multiple hops, so the VP and VC indices for the next hoop are written into the header of the cell as it 
passes through the switch 21 as indicated in Fig. 2 at 42. 

The switch 21 employs the VP and VC indices of the inbound cell (Fig. 2 at 41 ) to compute the address at which 
the connection record for the associated flow resides within the control RAM 29. Typically, this connection record 
includes a bit vector for identifying the output port (i.e., the switch-level "destination") at which the flow exits the switch 

40 21 , a priority index for identifying the relative priority of the flow on a granular priority scale, and a circuit index (Xircuit 
Index") which uniquely identifies the flow internally of the switch 21 . As shown in Fig. 2 at 43, these connection param- 
eters are written into the cell header. Then, the cell is written into the data RAM memory 29, while a pointer to the cell 
is linked into an appropriate one of a plurality of FIFO input queues, where the selection of the queue is based on the 
priority of the related flow. 

45 The relative priorities of the head of queue cells within these input queues are examined during each cell time, 

and the head of queue cell having the highest priority is selected for arbitration during the next arbitration session. 
Furthermore, the priority of any lower priority head of queue cell (i.e., any non-selected head of queue cell) is incre- 
mentally increased (by means not shown), thereby increasing the probability of that cell being selected for arbitration 
during the next arbitration session. Therefore, even though the higher priority input queues have greater throughput 

so per unit time than the lower priority queues, the lower priority queues have bounded delay because the priority of their 
head of queue cells increases as a function of time. 

Each arbitration cycle requires one cell time of the switch 21 . so the routing information for the cells that are selected 
for arbitration is fed into the reservation ring 26 one cell time prior to the release into the switching fabric 24 of the pay 
loads of the cell or ceils that win the arbitration. In other words, as shown in Fig. 2 at 44, the cells that are received by 

55 the reservation ring 31 and switching fabric 32 are composed of the headers of the cells for the next arbitration cycle 
(i.e., the "current cells") fol towed by the bodies or payloads of the celts successfully arbitrated during the previous 
arbitration cycle (i. e. , the "previous celts'). Thus, when the cell bodies reach the fabric 32, the fabric is already configured 
by the fabric control 33 to route those cells to their respective output port destination. For additional information on the 
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reservation ring 31 and the arbitration it performs, see Cisneros. K, "Urge Packet Switch and Detention Region 
JL. r „ ^ vm, motional S ^-hinn Symposium. 1990, paper 14, vol. III. pp. 77-83 and Lyles U. 

^^iXSS^Li may be proved tor testing the sw«ching processes) the cells are reaped 
£ Tqueuei J ^ he ita path, the contro. path, or both (a) for time scheduled transler to the appropriate output ports 
d fheTwSch 21 or (b) for transfer to the control processor module 22. The time scheduled transfer of cells to the 
2 ^ 1 4 switch 21 is central to this invention so that subject is discussed in further deta.l here nbelow On 
SeTh^d^ 

procMSor^S wim RM (Resource Management) and OAM (Operate and Maintenance) cel.s are madenta. top.cs 

iJJ^ut 2L on. output channel of the swftch 21 is shown, it wi« be understood that ft. channe. • general* 
^rs^ ten s^tTSLte module 51 for accept^ cel. bodies and their associated circuit indices from 

memoX^ This in turn enables the cel. flow control unit 55 to check the traffic sha P ,ng status of the ^ftow An OAW 
ZS^^ JLm^ is provided to enable the flow control unit 55 to identHy these control cell, and to 

to which it belongs. . M ^ nflr . vc aii Aues with respect to the depth 

log cells of such non-compliant flows. mftCSflnas , ha t are returned by the queue contro. unit 58 tocause 

Implement a staled vM clocK » M ^» =• •* ^ T.ZluWJUsns™^ M « ■——I <=«"«- 

. any. on .he pe.-VC qu.ee lor ». ~nne«,on MMM the "JJT f -i01fc ^ „, „ cB ,«, looo 
,ele,.ne. » the sohedele, 62 lo- eehrtiknt The.. ~ cIT*» 1 Sb> W «MUng 
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is designed to support to prevent time wrap induced ambiguities and preferably is twice as long so relative times can 
be compared using two's complement computations. 

B. Shaping Flows of Fixed Bit Length Data Transport Limits to Specified Peak Flow Rates 

5 

Referring to Fig. 4, it will be evident that stalled virtual clock transmission control is well suited for shaping time 
multiplexed flows of fixed bit length data transport units, such as ATM cells, from an output queued routing mechanism 
to specified peak data unit flow rates, such as PCRs for DBR/CBR ATM service. As previously described, the data 
transport units of the flows that are routed to a given output port are queued, after being routed, in per-flow queues. 

10 The data transport units at the heads of these queues then are admitted by an admission controller 61 (to the exclusion 
of all other transport units) for scheduling on a time line calendar queue 63 by a scheduler 62. The scheduler 62. in 
turn, performs per-flow virtual clock computations on these head of queue transport units to schedule them for release 
from the calendar queue 63 in accordance with their respective theoretical finish times, VT(f, j+1), or their respective 
theoretical start times. Please see Section III.C.4 above. 

'5 Real time advantageously is incremented on the time line 63 at a rate that enables the shaped, time multiplexed 

output traffic to essentially fill the bandwidth of the output link 71 . As will be recalled, the maximum number of resolvable 
time slots into which the scheduler 62 can map the head of queue members of the respective flows is based on the 
ratio of the maximum permissible frequency to the minimum permissible frequency of those flows. Thus, the rate at 
which real time is incremented from bucket-to-bucket at a rationale multiple of the cell rate. 

20 Data transport units residing in time slots which represent times that are earlier than or equal to the current real 

time of reference for the time line 63 are eligible for transmission and, therefore, are linked into a transport list 65 as 
previously described. However, those data transport units which reside in time slots that are associated with later time 
slots of the time line 63 remain in a pending state until system real time advances sufficiently to reach those time slots. 
To avoid rollover ambiguities, the time line 63 is designed to ensure that all references to earlier scheduled data transport 

2S limits are removed from each time slot before any references to later scheduled transport units are inserted therein in 
anticipation of the next scan. 

While the above-described arrangement effectively shapes conforming DBR/CBR ATM flows to the PCRs specified 
by their traffic contracts, it does not aid in bringing the Cell Delay variation (CDVs) of those flows into conformity with 
the t pcr parameters of their traffic contracts. 

30 

C. Multiple Priority Levels for Minimizing Relative CDV 

In accordance wit the present invention, data transport units that are delivered to a multiplexing point, such as an 
output port of an ATM switch, by flows having different frequencies are prioritized so that the data transport units of 

35 the higher frequency flows are given transmit priority over any data transport units of tower frequency flows with which 
they happen to collide. As shown in Fig. 3. this transmit priority can be implemented by steering the data transport 
units that are admitted for scheduling by a stalled virtual clock scheduling mechanism 63 or the like to one or another 
of a plurality of priority rank ordered time-lines 66a-66e or output FIFO queues based on the frequencies of the flows 
to which those respective data transport units belong. For example, for an ATM switch, it may be advisable to implement 

*o on the order of five different frequency dependent/class of service dependent output priorities, including (1 ) a top.priority 
for cells from flows that have negotiated output rates of at least 1/1 6 of the full rate of the output link (i.e. , its aggregate 
bandwidth), (2) a second priority for cells form flows having negotiated output rates ranging from 1/16 to 1/256 of the 
output link rate, and (3) a third priority for cells from flows having negotiated output rates ranging from 1/256*to 1/4096 
of the link rate. The lower two priorities then suitably are established for ABR connections that have non-zero negotiated 

4* MCR rates and for UBR connections and ABR connections that have MCR rates of 0, respectively. 

As will be appreciated the present invention effectively reduces the CDVs of the higher frequency flows, without 
materially increasing the CDVs of the lower frequency flows. As a general rule, the CDV that is tolerable is a function 
of the negotiated rate for a flow. For example, a CDV of 1 00 cell times is very large with respect to an expected emission 
interval of one cell every 10 cells, but generally insignificant if the negotiated emission interval is only one cell every 

so 2029 cells. 

When a calendar queue mechanism is employed to schedule the data transport units or cells of the different fre- 
quency flows lor transmission, the high frequency high priority flows need to be resolved to the precision of a single 
cell time while being scheduled to achieve an acceptably tow CDV, but the low frequency/low priority flows can be more 
coarsely resolved to a precision of, say, 16 cell times. This means that the number of time slots on the calendar queue 
55 63 can be reduced. This enables the amount of memory that is required to implement the calendar queue 63 to be 
reduced at the cost of losing some typically unneeded precision in the scheduling of the head of queue cells of the 
lower frequency flows. 

It is to be understood that the frequency based prioritization technique which the present invention provide for 
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resolving transmission conflicts at multiplex points among flows of different nominally fixed frequencies may be em- 
ployed in many different applcations for reducing the relative jitter of the flows, including in applications having work 
conserving per-flow output queues lor feeding cells or other data transport units into such a multiplex point. 

5 D. Traffic Shaping to MCR and PCR Parameter for ABR Service in Networks Having Source-tc-Destination 
ABR Control Loops 

As will be recalled the standard ABR traffic contract contemplates explicitly or implicitly specified PCR and MCR 
parameters, where MCR may equal 0. An ABR flow that conforms to its GCRA (1/MCR. t MCR ) is valid and entitled to 
w service, but the network may not be giving the flow the bandwidth to which rt is entitled. Contrasting*, a flow wh.ch 
violates its GCRA (1/PCR. Tpc R is non-conformng. Thus. ABR connections or flows that are routed to ' h< .output of a 
network or network element, such as by the ATM switch 21 , advantageous* are shaped to ensure. (1 } that the ABR 
flows with non-zero MCR guarantees receive adequate output bandwidth to effectively satisfy those guarantees, and 
(2) that none of the ABR flows violate their PCR commitments at such output. In networks that have multiple multiplex 
n noints it is useful to perform this shaping at each of the multiplex points. 

Fiq 4 illustrates one technique for shaping ABR flows or VC connections to conform them to their trafc contracts 
at an output of a network or network element. As shown, the rates at which the respective flows are requesting output 
serlice are monitored; first to determine at 85 whether the flows are or are no, GCRA (1/MCR, x MCR ) compliant, and 
then to determine at 86 whether the GCRA (1/MCR, W non^ompliant flows are or are not GCRA (1/PCR. Tp CR ) 
XL. Control signals reflecting the results of these cascaded tests 85 and 86 are fed back to he ^^62 
^ as associated steering logic 87 to permit approve rate shaping adjustments to be made to the .rates i at wh«h 
the data transport units (e.g., cells) of the respective flows are fed to the output As will be W^r*An%ln 
state is for all of the ABR flows to be GCRA (1/MCR, *mcr) non-compliant (or just barely compliant) ^^V^^' 
tpc) false/compliant because those algorithms test for minimum and maximum acceptable flow rates re p*t,ve* _ 
More oarticularlv GCRA (1/MCR, ^u) compliant flows may not be receiving the output bandwidth that they need 

to^st^M^ 
8 7 "es«-upto^ 

high parity stalled virtual ckxk calendar queue 88 for output at a rate somewhat in excess of rts guaranteed MCR 
For e^am pTe, these subsequent* received cell or packet references suitab* are scheduled onthe 
for emisston at a rate that is determined by multiplying the MCR for the flow by a predetermined speedup factor. This 
causTs Tnetwo^ to give these additional potent*.* "lagging- flows the additional output bandwidth that • needed 

t0 ^CRaTi /MCR* tmcr) non^o m P |iant ABR flows are ,urtner t9S,ed at 86 10 det9rrnine wne,her tne V are damandin 9 
outputse^eaS^ 

Svi* fteTesufts of L test are captured by another control signal which is fed backto the scheduler 62 an djs 
Tering togic 87 to set them up tor rate sensitive enqueuing of the subsequently receded references to these GCRA 
/MCR t") non-compliant flows on a lower priority queue 89. Specifically, the subsequent* received references 
S S ZrTo P i GCRA (1/PCR, W can** at 86 are appended to the tail ofthc ttvm** JrJ. 
ABR flows with MCR guarantees for it is serviced in round-robin order. Contrast.no*, when a flow s found to be GCRA 
n/PCR L a ) non-compliant at 86. the subsequent* received references for that flow are scheduled by the scheduler 
62oSano^trc^^^ 

im ^oTge P n^ 

If GCRA (1/MCR. Tmc R ) compliant then enqueue in high priority queue at rate MCFPspeedup 
else if GCRA (1/PCR, trcr) compliant then enqueue in low-priority work conserving queue 
else enqueue in low-priority non-work conserving queue based on PCR emisston interval. 

However, alternatives that are based on similar and different principles will suggest 

For example, as shown in Fig. 5, one potential* attractive attemative is to 'J^^J* *J 

transport units or cells of flows that are found to be GCRA (1/MCR, r MCR ) non-compliant and GCRA < 1 ^^CR> 
™ V an , at 65 and 86 (Fiq 4) respective*, in both (1) the high priority calendar queue 88 at a MCR scheduling 
inZarlnd (2) fte wo!k clerving Region of the lower priority queue. This creates a -race' conditton because when- 

me Xr instance of that reference from the other queue. This means that a tighter bound may be ma.ntamed on the 
tolerance, t. The traffic shaping algorithm of this modified embodiment is: 
At enqueue: 
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enqueue in high priority queue with rate MCR; 

if GCRA (1/PCR, tpcp) compliant then enqueue in taw-priority work conserving queue; 
else enqueue in low-priority non-work conserving queue based on PCR emission interval: 
At dequeue: 

$ if dequeuing form high priority queue then delink from low priority queue else delink from high priority queue. 

Fig. 6 illustrates stiil another alternative for shaping ABR connections or flows. In this embodiment, the output 
bandwidth that is provided for each MCR non-compliant ABR flow that has a non-zero MCR guarantee is dynamically 
adjusted based on the average spacing between ABR cells on the output link (i.e. , at the transmit side of a line interface) 

10 and the current length of the ABR "ready queue.' The current length, Len, of this ABR 'ready queue' convenientfy is 
determined by measuring the length of the work conserving queue on the ABR transmit lists 65e and 65f as at 93. On 
the other hand, the average spacing, S, between ABR cells on the calendar queue can be determining at 94 the percent 
of the link by dividing the bandwidth that is available for ABR flows after allowance is made for any non-ABR flows. 
The bandwidth required by these non-ABR flows suitably is determined by computing at 94 the aggregate bandwidth 

is needed to service the transmit list 65 entries for the non-ABR flows. 

For example, if it is determined at 94 that 50% of the bandwidth of the output link is available for the ABR flows, 
it follows that the average space, S, between ABR cells or data transport units on the output link then is two cell times. 
This information, together with the computed length, Len, of the ABR 'ready queue,' enables the service for the MCR 
non -compliant ABR flows to be optimized by having the scheduler 62 schedule the incoming references for those flows 

20 on the non-work conserving calendar queue 89 at respective emissions intervals T, of: 

T, = Min (S t , Max (P t , R)) (3) 

25 Where: 

R = an estimated round robin ABR 

service time (R = current time + S*Len); 
P t = the earliest service contract compliant departure for the next cell of flow t (P t = current time + 1/PCR); and 
30 S t = the latest service contract compliant departure for the next cell of flow t (S( = current time + 1/MCR) 

E. Traffic Shaping for Segmented ABR Connections 

As is known, ABR connections may operate with source-to-destination control loops, such as shown in Fig. 7 at 
55 101 , or with segmented control loops, such as shown in Fig. 8 at 102. When segmented control loops are employed, 
each of the separately controlled segments functions as a virtual source, VS, for the next segment. Thus, each ABR 
control segment (except the first) is sourced by a virtual source which assumes the behavior of an ABR source endpoint. 
Backward resource management (RM) cells are removed from the control loop when they are received by a virtual 
source. However, each source or virtual source can compute an estimate of the available network bandwidth to its 
40 destination or virtual destination, VD, based on this feedback which, in turn, can be expressed as an Allowed Cell Rate 
(ACR) on a per-segment basis. Therefore, ABR traffic shaping advantageously is carried out segment-by-segment for 
these ABR connections with segmented control loops, using a maximum allowable per-flow data transport unit or cell 
rate of Max(MCR, Min(ACR, PCR). The embodiments of Figs. 4-6 can implement such traffic shaping by setting PCR* 
equal to Max(MCR, Min(ACR, PCR) and then by using PCR* in place of PCR in the computations. 



Claims 

1 . A process for shaping time multiplexed serial flows of packets to respective available bit rate network traffic con- 
tracts in a network having a segmented control loop (102) which is configured to provide segment-by-segment 
congestion control for said network; said traffic contracts establishing respective minimum and peak packet emis- 
sion rates for said flows and respective tolerances on said rates; said process comprising 

analyzing congestion control activity within the respective segments of said control loop (102) for estimating 
how much bandwidth the respective segments of said network are able to provide for available bandwidth 
traffic: 

employing said bandwidth estimates to compute allowed packet rates for the respective available bit rate flows 
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